Assessing Ranking Fidelity of Competition Structures via Weighted Mutual Information
Good
Bad
Distribution of \(\hat{r}\)
Effectivity and Efficacy
Effectivity and Efficacy
- Salient ranks: output ranks of the tournament you want to get correct
- Effectivity: 1 is only salient rank (k=1)
- Efficacy: 1 through \(n\) are salient ranks (k=n)
Weighted Mutual Information
- A limitation of mutual information is that it does not distinguish between the direction of information
- Given equal probabilities, the information found from a ranking of \((1,2,3,4)\) is equivalent to \((4,3,2,1)\)
- Because of this, we proposed a new weighting function
- The final metric we used normalizes the results: \(\frac{I_w(r,\hat{r})}{H(r,\hat{r})}\)
Top K-teams Weighted Kendall
\[\tau_w = \frac{\sum_{i\neq j} W_{ij} \times \text{sgn}(i-j)\times \text{sgn}(R_i - R_j)}{\sum_{i,j} W_{ij} - \sum_i W_{ii}}\]
Tournament Simulations
- Given the assigned “known strength”, \(\theta\), we simulate a game using: \(\Pr(i > j) = \frac{e^{\theta_i}}{e^{\theta_i} + e^{\theta_j}} = \frac{1}{1+e^{\theta_i - \theta_j}}\)
| r | theta | r_hat | Wins | Losses |
|---|---|---|---|---|
| 1 | 1.221 | 3 | 2 | 1 |
| 2 | 0.765 | 4 | 1 | 2 |
| 3 | 0.431 | 2 | 2 | 1 |
| 4 | 0.140 | 1 | 3 | 0 |
| 5 | -0.140 | 7 | 0 | 1 |
| 6 | -0.431 | 6 | 0 | 1 |
| 7 | -0.765 | 5 | 0 | 1 |
| 8 | -1.221 | 8 | 0 | 1 |
Results
